Cloud-based disaster recovery of backup data and metadata

ABSTRACT

Cloud storage services can be used to facilitate secondary backup and disaster data recovery without the need for specialized backup servers at the secondary location or cloud storage service. Backup data streams are transferred to a cloud storage service. In addition to the backup data streams, backup metadata is generated for each backup data stream. The backup metadata is adapted to configure a backup server to retrieve and access data in the backup data stream. The backup metadata is also transferred to the cloud storage service. To access data from the backup data stream, a recovery backup system is connected with the cloud storage service. Backup metadata is transferred from the cloud storage service to the recovery backup system. The recovery backup system is updated with the backup metadata, which configures the recovery backup system to retrieve and access data in the backup data stream.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. Patent Application No. 61/290,334,filed Dec. 28, 2009 and entitled “DEDUPLICATED OBJECT STORAGE SYSTEM ANDAPPLICATIONS,” and U.S. Provisional Patent Application No. 61/315,392,filed Mar. 18, 2010 and entitled “WAN-OPTIMIZED LOCAL AND CLOUD SPANNINGDEDUPLICATED STORAGE SYSTEM,” which is incorporated by reference hereinfor all purposes. This application is related to U.S. patent applicationSer. No. ______ [Attorney Docket No. R001710US], filed ______ andentitled “CLOUD SYNTHETIC BACKUPS,” which is incorporated by referenceherein for all purposes.

BACKGROUND OF THE INVENTION

The present invention relates generally to data storage systems, andsystems and methods to improve storage efficiency, compactness,performance, reliability, and compatibility. Many data storage systemsare tasked with handling enormous amounts of data. To protect theirdata, many organizations use backup systems to store multiple copies ofimportant data on-site and/or off-site. Backup systems can createmultiple backup data sets and/or snapshots, enabling organizations tomaintain copies of their data at different time periods or instances.Backup systems can also create incremental backups, which record changesin an organization's data subsequent to a previous full or incrementalbackup data set.

Organizations often prefer to store at least some backup data sets at adifferent location than their primary data center. This protects theorganization's data from accidents or disasters occurring at the primarydata center. However, maintaining computers and data storage at multiplelocations is expensive and time-consuming. As an alternative, manyorganizations rely on a third-party to provide off-site data storage.Cloud storage services are one type of off-site data storage. Cloudstorage services are data storage services available via a wide-areanetwork. Cloud storage services provide storage to users in the form ofa virtualized storage device available via the Internet. In general,users access cloud storage to store and retrieve data using web servicesprotocols, such as REST or SOAP.

Cloud storage service providers manage the operation and maintenance ofthe physical data storage devices. Users of cloud storage can avoid theinitial and ongoing costs associated with buying and maintaining storagedevices. Cloud storage services typically charge users for consumptionof storage resources, such as storage space and/or transfer bandwidth,on a marginal or subscription basis, with little or no upfront costs. Inaddition to the cost and administrative advantages, cloud storageservices often provide dynamically scalable capacity to meet its userschanging needs.

Despite the cost and administrative advantages of cloud storageservices, integrating cloud storage services with backup systems can bechallenging. First, many backup systems require a specialized backupserver at each backup site to store and maintain backup data sets.However, cloud storage services often only provide a virtualized datastorage device to their users. Adding and maintaining a backup server atthe cloud storage site, for example as a physical server or within avirtual machine, increases the cost and complexity of the cloud storageof the cloud storage service. Additionally, because the wide-areanetwork typically has much lower bandwidth and higher latency thanlocal-area networks, access to backup data sets in the cloud storageservice is much slower. This can make some operations too slow to bepractical. For example, using cloud storage services to create asynthetic backup, which is a complete backup data set created by copyingdata from two or more previous backup data sets, including at least oneincremental backup data set, is extremely slow due to the performance ofthe wide-area network.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the drawings, inwhich:

FIG. 1 illustrates a typical prior redundant backup system;

FIG. 2 illustrates a cloud backup system suitable for use withembodiments of the invention;

FIG. 3 illustrates a method of performing disaster recovery using acloud backup system according to an embodiment of the invention;

FIG. 4 illustrates an example of a deduplicating cloud-spanning storageinterface suitable for use with embodiments of the invention;

FIG. 5 illustrates example data structures used by a deduplicatingcloud-spanning storage interface suitable for use with embodiments ofthe invention;

FIGS. 6A-6B illustrate a method of creating cloud synthetic backupsaccording to an embodiment of the invention; and

FIG. 7 illustrates a computer system suitable for implementingembodiments of the invention.

SUMMARY

An embodiment of the invention eliminates the need for specializedbackup servers at the secondary location, such as the secondary backupmaster policy server and secondary backup media server. This enablescloud storage to be used as inexpensive and easily configured andmaintained off-site data storage for backup systems. In an embodiment,backup data streams or other forms of backup data are transferred to acloud storage service. In addition to the backup data streams, backupmetadata is generated for each backup data stream. The backup metadatais adapted to configure a backup server to retrieve and access data inthe backup data stream. The backup metadata is also transferred to thecloud storage service.

To access data from the backup data stream, an embodiment of theinvention connects to the cloud storage service from a secondary orrecovery location. Backup metadata is transferred from the cloud storageservice to a recovery backup system at the secondary or recoverylocation. The recovery backup system is updated with the backupmetadata, which configures the recovery backup system to retrieve andaccess data in the backup data stream. In an embodiment, the recoverybackup system processes the backup metadata in order of its creation.

Embodiments of the invention transfer backup data streams and/or backupmetadata to the cloud storage service in deduplicated form. The backupdata stream and/or backup metadata is provided to the recovery locationin deduplicated form, where the corresponding undeduplicated versionsare reconstructed in whole or part. In an embodiment, a cloud spanningstorage interface is used to convert data between deduplicated andundeduplicated forms.

DETAILED DESCRIPTION

FIG. 1 illustrates a typical prior redundant backup system 100.Redundant backup system 100 includes a primary backup system 103, whichmay be located at an organization's data center. Primary backup system103 supports one or more client systems 105, such as client systems 105a, 105 b, and 105 c. Client systems 105 can include personal computers,application server computers, database servers, or any other type ofdevice or computer system storing data in an organization. The clientsystems 105 each include backup agent applications 110 that supervisethe transfer of data from the client systems 105 during data backups andto the client systems 105 when restoring data from a data backup.

Primary backup system 103 includes a backup master policy server 115.Backup master policy server 115 initiates data backups according topolicies and/or commands provided by system administrators. Backupmaster policy server 115 may initiate full or incremental data backupsfrom one or more client systems 105. Full data backups copy all or aspecified portion of the data of one or more client systems 105.Incremental data backups copy only the portion of data from a data setthat has changed since a previously completed full or incremental databackup.

Backup master policy server 115 maintains backup metadata describing thecontents of each data backup. Backup metadata can include the source ofthe data in the data backup, the time and data of the data backup, and,if the data backup is an incremental backup, the dependencies of thedata backup on previously completed data backups. In general, backupmetadata instructs the primary backup system 103, or any other backupsystem, how to use one or more backup data sets to restore data on oneor more client systems 105. Without backup metadata, the backup datasets cannot be used.

The primary backup system 103 also includes a backup media server 120.Backup media server 120 is adapted to receive data from one or moreclient systems 105 during one or more data backups, assemble thereceived data into one or more data backup files or backup data streams,and transfer the assembled data backup to one or more data storagedevices 125, such as local data storage devices 125 a and 125 b. Localdata storage devices 125 can include file servers, block-based storagedevices, and storage array networks. The backup media server 120 mayaccess data storage devices 125 using one or more file system protocols,such as CIFS or NFS, or block-based storage protocols, such as iSCSI oriFCP.

The primary backup system 103 may include one or more local-areanetworks, such as LAN 107, for facilitating communications betweenclient systems 105, the backup master policy server 115, and the backupmedia server 120. Additionally, LAN 107 may connect the backup mediaserver 120 with one or more of the storage devices 125. Optionally, aseparate storage array network SAN or LAN 109 may connect the backupmedia server 120 with one or more of the storage devices 125.

To protect data against accidents or disasters occurring at the locationof the primary backup system 103, redundant backup system 100 includes asecondary backup system 130 at a different physical location than theprimary backup system 103. Secondary backup system 130 includes asecondary backup master policy server 135, a backup media server 140,and one or more storage devices 145. Secondary backup system 130 iscapable of communicating with the primary backup system via a wide-areanetwork (WAN), such as the internet or a private WAN.

Typically, the secondary backup system 130 requires its own backupmaster policy server 135 and backup media server 140, because the backupmaster policy server 115 and backup media server 120 in the primarybackup system 103 are incapable communicating with storage outside ofthe primary backup system location. Thus, transferring data backups fromthe primary backup system 103 to the secondary backup system 130 is aconvoluted process.

For example, a backup data set created by the primary backup system 103may be stored on storage device 125 b. Storage device 125 b initiatesthe copying of the backup data set to storage device 145 a in thesecondary backup system 130. This transfer may use one or morelocal-area and/or wide-area networks.

Once the backup data set has copied to one of the storage devices 145 inthe secondary backup system 130, the secondary backup master policyserver 135 needs to be updated with backup metadata for the copiedbackup data set. Typically, the secondary backup master policy server135 is updated with backup metadata for a transferred backup data set bytransferring a copy of the backup metadata 150 from the primary backupsystem 103 to the secondary backup system 130. For example, a copy ofthe backup metadata 150 for a copied backup data set is transferred fromthe backup master policy server 115 to the storage device 125 b via thebackup media server 120. The storage device 125 b then transfers 155 thecopy of the backup metadata to the storage device 145 a. Storage device145 a transfers this copy of the backup metadata to the secondary backupmaster policy server 135 by transferring 160 the copy of the backupmetadata 150 from the storage device 145 a to the secondary backup mediaserver 140. Then, the copy of the backup metadata 150 is transferred 165from the secondary backup media server 140 to the secondary backupmaster policy server 135. The secondary backup master policy server 135uses the copy of the backup metadata to recognize the copied data backupset and make this copied data backup set available to restore data ifneeded.

An embodiment of the invention eliminates the need for specializedbackup servers at the secondary location, such as the secondary backupmaster policy server 135 and secondary backup media server 140. Thisenables cloud storage to be used as inexpensive and easily configuredand maintained off-site data storage for backup systems.

FIG. 2 illustrates a cloud backup system 200 suitable for use withembodiments of the invention. Cloud backup system 200 includes a primarybackup system 203, similar to primary backup system 103 discussed above.Primary backup system 203 may be located at an organization's datacenter. Primary backup system 203 supports one or more client systems205, such as client systems 205 a, 205 b, and 205 c. Client systems 205can include personal computers, application server computers, databaseservers, or any other type of device or computer system storing data inan organization. The client systems 205 each include backup agentapplications 210 that supervise the transfer of data from the clientsystems 205 during data backups and to the client systems 205 whenrestoring data from a data backup.

Primary backup system 203 includes a backup master policy server 215.Backup master policy server 215 initiates data backups according topolicies and/or commands provided by system administrators. Backupmaster policy server 215 may initiate full or incremental data backupsfrom one or more client systems 205.

Backup master policy server 215 maintains backup metadata describing thecontents of each data backup. Backup metadata can include the source ofthe data in the data backup, the time and data of the data backup, and,if the data backup is an incremental backup, the dependencies of thedata backup on previously completed data backups. In general, backupmetadata instructs the primary backup system 203, or any other backupsystem, how to use one or more backup data sets to restore data on oneor more client systems 205.

The primary backup system 203 also includes a backup media server 220.Backup media server 220 is adapted to receive data from one or moreclient systems 205 during one or more data backups, assemble thereceived data into one or more data backup files or backup data streams,and transfer the assembled data backup to one or more data storagedevices 225, such as a local data storage device 225 a. Local datastorage devices 225 can include file servers, block-based storagedevices, and storage array networks. The backup media server 220 mayaccess data storage devices 225 using one or more file system protocols,such as CIFS or NFS, or block-based storage protocols, such as iSCSI oriFCP.

The primary backup system 203 may include one or more local-areanetworks, such as LAN 207, for facilitating communications betweenclient systems 205, the backup master policy server 215, and the backupmedia server 220. Additionally, LAN 207 may connect the backup mediaserver 220 with one or more of the storage devices 225. Optionally, aseparate storage array network SAN 209 may connect the backup mediaserver 220 with one or more of the storage devices 225.

To protect data against accidents or disasters occurring at the locationof the primary backup system 203, cloud backup system 200 includes acloud storage service 250. Cloud storage service 250 is connected withthe primary backup system 203 via a wide-area network 230, such as theinternet or a private WAN. Cloud storage service 250 includes a cloudstorage interface 255 and data storage 260. Cloud storage interface 255receives data read and write requests using cloud storage protocols, forexample based on web services protocols such as SOAP or REST, andperforms corresponding storage operations with data storage 260.

In an embodiment, primary backup system 203 includes a cloud spanningstorage interface device 225 b. Cloud spanning storage interface 225 benables backup media server 220 to communicate with cloud storageservice 250 as if the service 250 was a local storage device. The cloudspanning storage interface 225 b stores and retrieves data from a cloudstorage service 250 via wide-area network 230 using cloud storageservice access protocols, such as SOAP or REST. The cloud spanningstorage interface 225 may provide an interface to the backup mediaserver 220 in the form of a file system protocol, such as CIFS or NFS, ablock-based storage protocols, such as iSCSI or iFCP, or a standard orproprietary API provided by the backup media server 220. In a furtherembodiment, the backup media server 220 uses an API to support one ormore storage plug-in software modules adapted to communicate with thecloud spanning storage interface 225 b.

Embodiments of the cloud spanning storage interface 225 b include alocal storage cache for storing data frequently accessed by the backupmedia server. Embodiments of the cloud spanning storage interface 225 bmay use local storage to queue data that is being transferred to thecloud storage service 250 via the wide-area network. Further embodimentsof the cloud spanning storage interface 225 b may predict data likely tobe requested by the backup media server 220 and prefetch and cache thisdata in local storage. In still further embodiments, the cloud spanningstorage interface 225 b may perform data deduplication and/or datacompression to reduce the amount of data storage required to store dataand to maximize the use of available bandwidth of the wide-area network230 in transferring data between the primary backup system 203 and thecloud storage service 250.

As shown in FIG. 2, system 200 does not require the use of specializedbackup servers at the secondary storage location, which in this exampleis the cloud storage service 250. FIG. 3 illustrates a method 300 ofperforming disaster recovery using a cloud backup system according to anembodiment of the invention.

Step 305 receives a data backup from a backup media server and transfersthe data backup to a cloud storage service. The data backup may includeone or more backup data files and/or one or more one or more raw streamsof backup data. Step 305 may optionally maintain a local copy of some orall of the transferred data backup.

Step 310 generates backup metadata for the received data backup.Embodiments of step 310 may receive backup metadata for the receiveddata backup from a master policy backup server in the primary backupsystem. Alternate embodiments of step 310 may query or access the masterpolicy backup server or its associated data to retrieve backup metadata.

In an embodiment, backup metadata is in a format adapted to be read byone or more additional master policy backup servers to recognize thecopied data backup set and make this copied data backup set available torestore data if needed. For example, backup metadata may be provided ina portable data format.

In a further embodiment, backup metadata is in the form of a log file orother data structure that indicates the completion of one or more databackup and/or replications of data backups to a secondary backup system.In yet a further embodiment, a cloud spanning storage interface mayprovide the primary backup system with an indication that a backup dataset has been replicated by a secondary backup system, even though thecloud spanning storage interface has actually only transferred the databackup to the cloud storage. The cloud spanning storage interface thenreceives or retrieves a backup replication completion log file from theprimary backup system to be used as backup metadata.

Step 315 transfers the backup metadata to the cloud storage service forstorage. In an embodiment, step 315 stores the backup metadata in thecloud storage separately from the data backup, such as in a differentstorage location or associated with a different storage identifier. Inan embodiment, the backup metadata is stored in the cloud storage in amanner such that an association between the backup metadata and the databackup is apparent.

Steps 305, 310, and 315 may optionally be repeated any arbitrary numberof times to transfer additional data backups and associated backupmetadata from the primary backup system to the cloud storage.

If there is a need to restore data from one or more data backups using adifferent backup system, such as following an accident, disaster, orother catastrophic failure at the location of the primary backup system,step 320 connects a new backup system, referred to as the recoverysystem, to the cloud storage service. The recovery system includes arecovery master policy server, a recovery media server, and a recoverycloud spanning storage interface, similar to their counterparts in theprimary backup system.

In step 325, the recovery system retrieves the backup metadata for thedata backups stored in the cloud storage. In an embodiment, uponconnecting with the cloud storage service, the recovery cloud spanningstorage interface searches for and retrieves all of the backup metadatathe cloud storage service.

Following the retrieval of the backup metadata from the cloud storageservice in step 325, step 330 updates the recovery master policy serverin the recovery system using the retrieved backup metadata. In anembodiment of step 330, the recovery cloud spanning storage interfaceprovides backup metadata to the recovery master policy server in thesame order that the backup metadata was created. In response to each setof backup metadata, the recovery master policy server updates itsinternal data structures so that it recognizes the associated databackup stored in the cloud storage service.

In a further embodiment, the backup metadata is in the form of a logfile or other data structure that indicates the completion of one ormore data backup and/or replications of data backups to a secondarybackup system. An embodiment of step 330 “replays” operations,transactions, or other events described in the backup metadata log fileor other data structure to configure the recovery master policy server.For example, a cloud spanning storage interface in the recovery systemmay use an API to inform the recovery master policy server that the databackups identified in the backup metadata are complete. In this example,even though the recovery master policy server did not initiate the databackups identified in the backup metadata (because they were performedby the primary backup system), replaying the operations, transactions,or other events in the backup metadata log file will configure therecovery master policy server appropriately. This embodiment using databackup or replication log files or other data structures as backupmetadata allow for the implementation of cloud-based disaster recoverysystems without explicit support for disaster recovery or cloud-basedstorage by the backup system.

Following the processing of all of the backup metadata retrieved fromthe cloud storage service, the recovery master policy server isconfigured to access any of the available data backups in the cloudservice. Following step 330, the recovery system may restore data fromone or more data backups stored by the cloud storage service. Forexample, a system administrator may initiate a data recovery byselecting data to restore from a list of data backups provided by therecovery master policy server of the recovery system. The recoverymaster policy server can provide this list of data backups because itsinternal data has been configured with backup metadata as describedabove. In response to a selection of data to restore, the recovery mediaserver requests the appropriate data backups from the recovery cloudspanning storage interface. The recovery cloud spanning storageinterface retrieves the requested data backups from the cloud storageservice. The recovery media server uses the retrieved data backups torestore the requested data.

It should be noted that there is no limit to the amount of time that maypass between the transfer of backup metadata to the cloud storageservice and the updating of the recovery system with the backupmetadata. Provided that backup metadata is processed in the correctorder when it is eventually retrieved, backup metadata stored in thecloud storage service is still valid months or years following itscreation.

The term “data deduplication” refers to some process of eliminatingredundant data for the purposes of storage or communication. Datadeduplicating storage typically compares incoming data with the dataalready stored, and only stores the portions of the incoming data thatdo not match data already stored in the data storage system. Datadeduplicating storage maintains metadata to determine when portions ofdata are no longer in use by any files or other data entities.

In an optional embodiment, the cloud spanning storage interface performsdata deduplication to reduce the amount of cloud storage required tostore data backups, maximize data transfer performance via a wide-areanetwork with the cloud storage service, and, as discussed in detailbelow, efficiently create synthetic backups from backup data stored incloud storage services.

Examples of deduplicating cloud spanning storage interfaces aredescribed in detail in U.S. Provisional Patent Application No.61/315,392, filed Mar. 18, 2010 and entitled “WAN-OPTIMIZED LOCAL ANDCLOUD SPANNING DEDUPLICATED STORAGE SYSTEM,” which is incorporated byreference herein for all purposes. FIGS. 4 and 5 summarize the operationof an example cloud spanning storage interface suitable for use withembodiments of the invention.

FIG. 4 illustrates an example of cloud spanning storage interface 400according to an embodiment of the invention. Cloud spanning storageinterface 400 includes one or more client systems 405, which may includeclient computers, server computers, and standalone network devices.Client systems 405 are connected with a cloud spanning storage interface425 via a local-area network and/or a storage area network 415. Cloudstorage 475 is connected with the cloud spanning storage interface 425by at least a wide-area network 477 and optionally an additional localarea network. Cloud storage 475 includes a cloud storage interface 480for communicating with the cloud spanning storage interface 425 viawide-area network 477 and at least one physical data storage device 485for storing data.

Embodiments of cloud spanning storage interface 400 may support avariety of different storage applications using cloud data storage,including general data storage, data backup, disaster recovery, anddeduplicated cloud data storage. In the case of general data storageapplications, a client, such as client 405 c, may communicate with thecloud spanning storage interface 425 via a file system protocol, such asCIFS or NFS, or a block-based storage protocol, such as iSCSI or IFCP.Data backup and disaster recovery applications may also use theseprotocols or specific backup and recovery protocols, such as VTL or OST.For backup applications, a client system 405 a may include a backupagent 410 for initiating data backups. The backup agent 410 maycommunicate directly with the cloud spanning storage interface 425 or abackup server 405 b, which in cloud spanning storage interface 400 isequivalent to a client. For cloud storage applications, a client 403 cmay communicate with the cloud spanning storage interface 425 via a webservices protocol, such as SOAP or REST. The web services protocol maypresent a virtualized storage device to client 403 c. The web servicesprotocol used by clients 405 to communicate with the cloud spanningstorage interface 425 may be the same or different than the protocolused by the cloud spanning storage interface 425 to communicate with thecloud storage 475.

Embodiments of the cloud spanning storage interface 400 may optimizedata access to cloud storage 475 in a number of different ways. Anembodiment of the cloud spanning storage interface 425 may presentclients 405 with a file system, backup device, storage array, or otherdata storage interface, while transparently storing and retrieving datausing the cloud storage 475 via the wide-area network 477. In a furtherembodiment, the cloud spanning storage interface 425 may perform datadeduplication on data received from clients 405, thereby reducing theamount of storage capacity required in cloud storage 475. Additionally,because the bandwidth of the wide-area network is often limited, datadeduplication by the cloud spanning storage interface 425 increases thedata access performance, as perceived by the clients 425. In still afurther embodiment, the cloud spanning storage interface 425 may locallycache a portion of the clients' data using local storage 470. Thelocally cached data may be accessed rapidly, further improving theperceived data access performance. As described in detail below, thecloud spanning storage interface 425 may use a variety of differentcriteria for selecting the portion of the clients' data to cache locallyand may locally cache data in a deduplicated form to reduce the requiredcapacity of local storage 475.

An embodiment of cloud spanning storage interface 425 includes one ormore front end interfaces 430 for communicating with one or more clientsystems 405. Examples of front end interfaces 430 include a backup frontend interface 430 a, a file system front end interface 430 b, a cloudstorage front end interface 430 c, a file archival front end interface430 d, and a object front end interface 430 e. An example backup frontend interface 430 a enables backup applications, such as a backup agent410 and/or a backup server 405 b, to store and retrieve data to and fromthe cloud storage 475 using data backup and recovery protocols such asVTL or OST. In this example, the backup front end interface 430 a allowsthe cloud spanning storage interface 425 and cloud storage 475 to appearto clients 405 as a backup storage device.

An example file system front end interface 430 b enables clients 405 tostore and retrieve data to and from the cloud data storage 475 using afile system protocol, such as CIFS or NFS, or a block-based storageprotocol, such as iSCSI or IFCP. In this example, the file system frontend interface 430 b allows the cloud spanning storage interface 425 andcloud storage 475 to appear to clients 405 as one or more storagedevices, such as a CIFS or NFS storage volume or a iSCSI or FibreChannellogical unit number (LUN).

An example cloud storage front end interface 430 c enables clients 405to store and retrieve data to and from the cloud data storage 475 usinga cloud storage protocol or API. Typically, cloud storage protocols orAPIs are implemented using a web services protocol, such as SOAP orREST. In this example, the cloud storage front end interface 430 callows the cloud spanning storage interface 425 and cloud storage 475 toappear to clients 405 as one or more cloud storage services. By usingcloud spanning storage interface 425 to provide a cloud storageinterface to clients 405, rather than letting clients 405 communicatedirectly with the cloud storage 475, the cloud spanning storageinterface 425 may perform data deduplication, local caching, and/ortranslation between different cloud storage protocols.

An example file archival front end interface 430 d enables clients 405to store and retrieve file archives. Clients 405 may use the cloudspanning storage interface 425 and the cloud storage 475 to store andretrieve files or other data in one or more archive files. The filearchival front end interface 430 d allows clients 405 to store archivefiles using cloud storage 475 using archive file interfaces, rather thana cloud storage interface. Additionally, the cloud spanning storageinterface 425 may perform data deduplication and local caching of thefile archives.

An example object front end interface 430 e enables clients to store andretrieve data in any arbitrary format, such as object formats and blobsor binary large objects. The object front end interface 430 e allowsclients 405 to store data in arbitrary formats, such as object formatsor blobs, using cloud storage 475 using object protocols, such as objectserialization or blob storage protocols, rather than a cloud storageprotocol. Additionally, the cloud spanning storage interface 425 mayperform data deduplication and local caching of the object or blob data.

In an embodiment, cloud spanning storage interface 425 also includes oneor more shell file systems 445. Shell file system 445 includes arepresentation of all of the entities, such as files, directories,objects, blobs, and file archives, stored by clients 425 via the frontend interfaces 430. In an embodiment, the shell file system 445 includesall of the entities stored by the clients 425 in a shell form. In thisembodiment, each entity, such as a file or other entity, is arepresented by a “shell” entity that does not include the data contentsof the original entity. For example, a shell file in the shell filesystem 445 includes the same name, file path, and file metadata as theoriginal file. However, the shell file does not include the actual filedata, which is stored in the cloud storage 475. It should be noted thatalthough the size of the shell file is less than the size of the actualstored file (in either its original or deduplicated format, anembodiment of the shell file system 445 sets the file size metadataattribute of the shell file to the size of the original file. In afurther embodiment, each entity in the shell file system 445, such as afile, directory, object, blob, or file archive, may include additionalmetadata for use by the cloud spanning storage interface 425 to accessthe corresponding data from the cloud storage 475.

An embodiment of the cloud spanning storage interface 425 includes adeduplication module 450 for deduplicating data received from clients405. Deduplication module 450 analyzes data from clients 405 andcompares incoming data with previously stored data to eliminateredundant data for the purposes of storage or communication. Datadeduplication reduces the amount of storage capacity used by cloudstorage 475 to store clients' data. Also, because wide-area network 477typically has bandwidth limitations, the reduction of data size due todata deduplication also reduces the amount of time required to transferdata between clients 405 and the cloud storage 475. Additionally,deduplication module 450 retrieves deduplicated data from the cloudstorage 475 and converts it back to its original form for use by clients405.

In an embodiment, deduplication module 450 performs data deduplicationon incoming data and temporarily stores this deduplicated data locally,such as on local storage 470. Local storage 470 may be a physicalstorage device connected with or integrated within the cloud spanningstorage interface 425. Local storage 470 is accessed from cloud spanningstorage interface 425 by a local storage interface 460, such as aninternal or external data storage interface, or via a local-areanetwork.

In an embodiment, the cloud storage 475 includes a complete andauthoritative version of the clients' data. In a further embodiment, thecloud spanning storage interface 425 may maintain local copies of someor all of the clients' data for the purpose of caching. In thisembodiment, the cloud spanning storage interface 425 uses the localstorage 470 to cache client data. The cloud spanning storage interface425 may cache data in its deduplicated format to reduce local storagerequirements or increase the effective cache size. In this embodiment,the cloud spanning storage interface 425 may use a variety of criteriafor selecting portions of the deduplicated client data for caching. Forexample, if the cloud spanning storage interface 425 is used for generalfile storage or as a cloud storage interface, the cloud spanning storageinterface may select a specific amount or percentage of the client datafor local caching. In another example, the data selected for localcaching may be based on usage patterns of client data, such asfrequently or recently used data. Caching criteria may be based onelapsed time and/or the type of data. In another example, the cloudspanning storage interface 425 may maintain locally cached copies of themost recent data backups from clients, such as the most recent fullbackup and the previous week's incremental backups.

In an embodiment, replication module 455 transfers locally storeddeduplicated data from the cloud spanning storage interface 425 to thecloud storage 475. Embodiments of the deduplication module and thereplication module 455 may operate in parallel and/or asynchronously, sothat the bandwidth limitations of wide-area network 477 do not interferewith the throughput of the deduplication module 450. The operation ofembodiments of deduplication module 450 and replication module 455 aredescribed in detail below.

An embodiment of cloud spanning storage interface 425 includes a cloudstorage backend interface 465 for communicating data between the cloudspanning storage interface 425 and the cloud storage 475. Embodiments ofthe cloud storage backend interface 465 may use cloud storage protocolsor API and/or web services protocols, such as SOAP or REST, to store andretrieve data from the cloud storage 475. In an embodiment, thereplication module transfers deduplicated data from local storage 470 tocloud storage 475 using the cloud storage backend interface 465. In anembodiment, the deduplication module retrieves deduplicated data fromthe cloud storage 475 using the cloud storage backend interface 465.

In an embodiment, the cloud spanning storage interface 425 performs datadeduplication by segmenting an incoming data stream to aid datacompression. For example, segmentation may be designed to produce manyidentical segments when the data stream includes redundant data.Multiple instances of redundant data may be represented by referencing asingle copy of this data.

Additionally, a data stream may be segmented based on data types to aiddata compression, such that different data types are in differentsegments. Different data compression techniques may then be applied toeach segment. Data compression may also determine the length of datasegments. For example, data compression may be applied to a data streamuntil segment boundary is reached or the segment including thecompressed data reaches a predetermined size, such as 4 KB. The sizethreshold for compressed data segments may be based on optimizing diskor data storage device access.

Regardless of the technique used to segment data in the data stream, theresult is a segmented data stream having its data represented assegments. In some embodiments of the invention, data segmentation occursin memory and the segmented data stream is not written back to datastorage in this form. Each segment is associated with a label. Labelsare smaller in size than the segments they represent. The segmented datastream is then replaced with deduplicated data in the form of a labelmap and segment storage. Label map includes a sequence of labelscorresponding with the sequence of data segments identified in thesegmented data stream. Segment storage includes copies of the segmentlabels and corresponding segment data. Using the label map and the datasegment storage, a storage system can reconstruct the original datastream by matching in sequence each label in a label map with itscorresponding segment data from the data segment storage.

Embodiments of the invention attempt (but do not always succeed) inassigning a single label to each unique data segment. Because thesegmentation of the data stream produces many identical segments whenthe data stream includes redundant data, these embodiments allow asingle label and one copy of the corresponding segment data to representmany instances of this segment data at multiple locations in the datastream. For example, a label map may include multiple instances of agiven label at different locations. Each instance of this labelrepresents an instance of the corresponding segment data. Because thelabel is smaller than the corresponding segment data, representingredundant segment data using multiple instances of the same labelresults in a substantial size reduction of the data stream.

FIG. 5 illustrates example data structures 500 used by a cloud spanningstorage interface according to an embodiment of the invention. Anembodiment of cloud spanning storage interface 500 includes both memory505, which has high performance but relatively low capacity, and diskstorage 510, which has high capacity but relatively low performance.

Memory 505 includes a slab cache data structure 515. The slab cache 515is adapted to store a set of labels 520 and a corresponding set of datasegments 525. In typical applications, the sets of labels 520 and datasegments 525 stored in the slab cache 515 represent only a smallfraction of the total number of data segments and labels used torepresent stored data. A complete set of the labels and data segments isstored in disk storage 510.

An embodiment of the slab cache 515 also includes segment metadata 530,which specifies characteristics of the data segments 525. In anembodiment, the segment metadata 530 includes the lengths of the datasegments 525; hashes or other characterizations of the contents of thedata segments 525; and/or anchor indicators, which indicate whether aparticular data segment has been designated as a representative exampleof the contents of a data segment slab file, as discussed in detailbelow.

An embodiment of the slab cache 515 also includes data segment referencecount values. The cloud spanning storage interface 500 recognizes thatsome data segments are used in multiple places in one or more datastreams. For at least some of the data segments, an embodiment of thecloud spanning storage interface 500 maintains counts, referred to asreference counts, of the number of times these data segments are used.As discussed in detail below, if a data stream includes a data segmentpreviously defined, an embodiment of the cloud spanning storageinterface 500 may increment the reference count value associated withthis data segment. Conversely, if a data stream is deleted from thecloud spanning storage interface 500, an embodiment of the cloudspanning storage interface 500 may decrement the reference count valuesassociated with the data segments included in the deleted data stream.If the reference count value of a data segment drops to zero, the datasegment and label may be deleted and its storage space reallocated.

In addition to the slab cache 515, an embodiment of the cloud spanningstorage interface 500 includes a reverse map cache 540. In anembodiment, the reverse map cache 540 maps the contents of a datasegment to a label, for the labels stored in the slab cache 515. In anembodiment, a hashing or other data characterization technique isapplied to segment data. The resulting value is used as an index in thereverse map cache 540 to identify an associated label in the slab cache515. If the hash or other value derived from the segment data matches anentry in the reverse map cache 540, then this data segment has beenpreviously defined and is stored in the slab cache 515. If the hash orother value derived from the segment data does not match any entry inthe reverse map cache 540, then this data segment is not currentlystored in the slab cache 515. Because the slab cache 515 only includes aportion of the total number of labels used to represent data segments, adata segment that does not match a reverse map cache entry may eitherhave not been previously defined or may have been previously defined butnot loaded into the slab cache 515.

In an embodiment, memory 505 of the cloud spanning storage interface 500also includes an anchor cache 545. Anchor cache 545 is similar toreverse map cache 540; however, anchor cache 545 matches the contents ofdata segments with representative data segments in data segment slabfiles stored on disk storage 510. A complete set of data segments arestored in one or more data segment slab files in disk storage 510. In anembodiment, one or more representative data segments from each datasegment slab file are selected by the cloud spanning storage interface500. The cloud spanning storage interface 500 determines hash or otherdata characterization values for these selected representative datasegments and stores these values along with data identifying the file ordisk storage location including this data segment in the anchor cache545. In an embodiment, the data identifying the file or disk storagelocation of a representative data segment may be its associated label.The cloud spanning storage interface 500 uses the anchor cache 545 todetermine if a data segment from a data stream matches a data segmentfrom another data stream previously stored in disk storage but notcurrently stored in the slab cache.

In an embodiment, potential representative data segments are identifiedduring segmentation of a data stream. As discussed in detail below, whenone or more potential representative data segments are later stored indisk storage 510, for example in a data segment slab file, an embodimentof the cloud spanning storage interface 500 selects one or more of thesepotential representative data segments for inclusion in the anchorcache.

A variety of criteria and types of analysis may be used alone ortogether in various combinations to identify representative datasegments in data streams and/or in data segment slab files stored indisk storage 510. For example, the cloud spanning storage interface 500selects the first unique data segment in a data stream as arepresentative data segment. In another example, the cloud spanningstorage interface 500 uses the content of the data stream to identifypotential representative data segments. In still another example, thecloud spanning storage interface 500 uses criteria based on metadatasuch as a file type, data type, or other attributes provided with a datastream to identify potential representative data segments. For example,data segments including specific sequences of data and/or located atspecific locations within a data stream of a given type may bedesignated as representative data segments based on criteria orheuristics used by the cloud spanning storage interface 500. In afurther example, a random selection of unique segments in a data streamor a data segment slab file may be designated as representative datasegments. In yet a further example, representative data segments may beselected at specific locations of data segment slab files, such as themiddle data segment in a slab file.

Disk storage 510 stores a complete set of data segments and associatedlabels used to represent all of the data streams stored by cloudspanning storage interface 500. In an embodiment, disk storage 510 maybe comprised of multiple physical and/or logical storage devices. In afurther embodiment, disk storage 510 may be implementing using a storagearea network.

Disk storage 510 includes one or more data segment slab files 550. Eachdata segment slab file 550 includes a segment index 555 and a set ofdata segments 565. The segment index 555 specifies the location of eachdata segment within the data segment slab file. Data segment slab file550 also includes segment metadata 560, similar to the segment metadata530 discussed above. In an embodiment, segment metadata 560 in the datasegment slab file 550 is a subset of the segment metadata in the slabcache 515 to improve compression performance. In this embodiment, thecloud spanning storage interface 500 may recompute or recreate theremaining metadata attribute values for data segments upon transferringdata segments into the slab cache 515.

Additionally, data segment slab file 550 may include data segmentreference count values 570 for some or all of the data segments 565. Inan embodiment, slab file 550 may include slab file metadata 575, such asa list of data segments to be deleted from the slab file 550.

Disk storage 510 includes one or more label map container files 580.Each label map container file 580 includes one or more label maps 590.Each of the label maps 590 corresponds with all or a portion of adeduplicated data stream stored by the cloud spanning storage interface500. Each of the label maps 590 includes a sequence of one or morelabels corresponding with the sequence of data segments in all or aportion of a deduplicated data stream. In an embodiment, each label mapalso includes a label map table of contents providing the offset orrelative position of sections of the label map sequence with respect tothe original data stream. In one implementation, the label maps arecompressed in sections, and the label map table of contents providesoffsets or relative locations of sections of the label map sequencerelative to the uncompressed data stream. The label map table ofcontents may be used to allow random or non-sequential access to adeduplicated data stream.

Additionally, label map container file 580 may include label mapcontainer index 585 that specifies the location of each label map withinthe label map container file.

In an embodiment, label names are used not only identify data segments,but also to locate data segments and their containing data segment slabfiles. For example, labels may be assigned to data segments duringsegmentation. Each label name may include a prefix portion and a suffixportion. The prefix portion of the label name may correspond with thefile system path and/or file name of the data segment slab file used tostore its associated segment. All of the data segments associated withthe same label prefix may be stored in the same data segment slab file.The suffix portion of the label name may be used to specify the locationof the data segment within its data segment slab file. The suffixportion of the label name may be used directly as an index or locationvalue of its data segment or indirectly in conjunction with segmentindex data in the slab file. In this implementation, the complete labelname associated with a data segment does not need to be stored in theslab file. Instead, the label name is represented implicitly by thestorage location of the slab file and the data segment within the slabfile. In a further embodiment, label names are assigned sequentially inone or more namespaces or sequences to facilitate this usage.

An embodiment similarly uses data stream identifiers to not onlyidentify deduplicated data streams but to locate label maps and theircontaining label map containers. For example, a data stream identifieris assigned to a data stream during deduplication. Each data streamidentifier name may include a prefix portion and a suffix portion. Theprefix portion of the data stream identifier may correspond with thefile system path and/or file name of the label map container used tostore the label map representing the data stream. The suffix portion ofthe data stream identifier may be used to directly or indirectly specifythe location of the label map within its label map container file. In afurther embodiment, data stream identifiers are assigned sequentially inone or more namespaces or sequences to facilitate this usage.

Embodiments of the cloud spanning storage interface 500 may specify thesizes, location, alignment, and optionally padding of data in datasegment slab files 550 and label map container files 580 to optimize theperformance of disk storage 510. For example, segment reference countsare frequently updated, so these may be located at the end of the datasegment slab file 550 to improve update performance. In another example,data segments may be sized and aligned according to the sizes andboundaries of clusters or blocks in the disk storage 510 to improveaccess performance and reduce wasted storage space.

A synthetic backup is a complete backup data set created by copying datafrom two or more previous backup data sets, including one full backupdata set followed by one or more incremental backup data set. Forexample, if a complete backup data set was created on June 1 and anincremental data backup was created on June 2, a synthetic backuprepresenting the complete backup data set on June 2 may be created bycombining these two backups. The incremental data backup only includesdata that has been added or changed since the complete backup data setwas created. The synthetic data backup can be created by combining thecontents of the incremental data backup with unchanged portions of thecomplete backup data set. If the synthetic backup is created frommultiple incremental backups, then the synthetic backup includes themost recent version of each portion of data in the backup data, asselected from one of the backup data sets.

A cloud synthetic backup is a synthetic backup created from two or moreprevious backup data sets, including at least one incremental backupdata set, stored in a cloud storage service. As described in detailbelow, an embodiment of the invention can create cloud synthetic backupswithout retrieving backup data from the cloud storage service ortransferring additional backup data to the cloud storage service.Further embodiments of the invention may also be used to createsynthetic backups from locally-stored deduplicated backup data alone orin combination with deduplicated backup data stored by a cloud storageservice.

FIGS. 6A-6B illustrate a method of creating cloud synthetic backupsaccording to an embodiment of the invention. FIG. 6A illustrates amethod 600 of creating and storing full and incremental data backup setsin a cloud storage service, enabling them to be used to create cloudsynthetic backups. Step 605 receives a backup data stream. The backupdata stream may be a raw data stream or one or more backup data filesprovided by a backup media server. The backup data stream may representa full or an incremental data backup. Step 610 performs datadeduplication on the received backup data stream to create datasegments, labels, and a label map corresponding with the received backupdata stream. Step 615 transfers the deduplicated backup data stream, inthe form of data segments, labels, and a label map to the cloud storageservice. In an embodiment, step 615 may transfer the deduplicated backupdata stream in the form of one or more data segment slab files and labelmap container files. Steps 605, 610, and 615 may be repeated formultiple full or incremental data backups created by the cloud backupsystem.

Following one or more iterations of steps 605, 610, and 615, the cloudstorage service includes one or more backup data sets, stored indeduplicated form as data segments, labels, and label maps. These backupdata sets may be used by an embodiment of the invention to create one ormore synthetic backups in the cloud storage service without retrievingbackup data from the cloud storage service, transferring additionalbackup data to the cloud storage service, or performing intensive datamanipulation on data in the cloud storage service.

FIG. 6B illustrates a method 650 of creating cloud synthetic backupsaccording to an embodiment of the invention. Step 655 receives a requestto initiate a synthetic backup. The request may be received from amaster policy backup server or a backup media server in response to anadministrator command or in response to a policy or criteria specifiedby an administrator.

To generate a synthetic backup, the backup media server identifies theportions of previously created backup data sets that need to be copiedto the new synthetic backup. Additionally, the backup media serveridentifies the locations or sequence of portions from previously createdbackup data sets to be copied to the new synthetic backup.

Step 660 receives a specification of a portion of a previously createdbackup to be copied to the new synthetic backup from the backup mediaserver. In an embodiment, this portion to be copied is specified byidentifying the previous backup data stream; the location, such as anaddress or offset, of the beginning of this portion; the size or endinglocation of this portion; and the destination location, such as anaddress or offset, of the portion in the synthetic backup. Alternateembodiments may specify the portion of a previously created backup to becopied to the new synthetic backup in a variety of different ways. Forexample, the destination location may be omitted if the backup mediaserver specifies portions to be copied sequentially. In this example,the destination location is implicitly defined as immediately followingthe previously specified portion.

Step 665 identifies one or more data segments corresponding with thespecified portion. In an embodiment, step 665 accesses the label map forthe previously created backup data set including the specified portion.Step 665 may retrieve copies of label maps as needed from the cloudstorage service or local copies of label maps after deduplicating data.In an embodiment, each label map includes metadata, such as data segmentsizes or data segment locations, for each label. Thus, step 665 is ableto identify a data segment corresponding with a given location in apreviously generated backup data stream. Based on the starting andending locations of the specified portion of a backup data set, step 665can identify corresponding labels in the label map.

Step 670 adds the identified labels to a new label map representing thesynthetic backup. Step 670 adds the identified labels to the new labelmap according to the destination location specified by the backup mediaserver. This may require inserting the identified labels betweenpreviously added labels if the backup media server specifies an explicitdestination location or appending the identified labels to the end of asequence of previously added labels if the backup media server specifiesportions sequentially.

Often, the portions of one or more previously created backup to becopied to a new synthetic backup may not correspond with the boundariesof corresponding data segments. For example, the portion of thepreviously created backup may have a starting address or offset afterthe beginning of a data segment and/or an ending address, offset orlocation before the end of a data segment. In this case, an embodimentof step 670 creates a label map for the synthetic backup that includesindicators specifying the use of partial data segments. For example, alabel and its optional indicators in a label map specify startingaddresses or locations and/or ending addresses or locations of data tobe included from their associated data segments. If a data segment is tobe included in its entirety in a synthetic backup, these indicators maybe omitted.

Steps 660, 665, and 670 may be repeated for an arbitrary number ofiterations in response to the backup media server specifying additionalportions of one or more backup data sets to be copied to the syntheticbackup. Once the backup media server has specified all of the portionsof one or more backup media sets to be copied to create the syntheticbackup, step 675 transfers the label map representing the syntheticbackup to the cloud storage.

Because this embodiment of the invention stores backup data in asegmented and deduplicated form, there is no need to directly copy orotherwise access the backup data or corresponding data segments frompreviously created backup data sets. Instead, the creation of thesynthetic backup can be performed as a manipulation of labels and labelmaps.

Furthermore, because all of the previously created backup data sets havebeen stored in the cloud storage service, there is no need to transferany additional backup data, apart from the new label map, to the cloudstorage service. Typical label maps are several orders of magnitudesmaller than the actual backup data. Thus, steps 665 through 675,including the transferring of the new label map to the cloud storageservice, can be performed very quickly.

Following the completion of the deduplicated synthetic backup, anembodiment of the invention provides a data stream identifier or one ormore shell files representing the deduplicated synthetic backup to thebackup system and/or any other storage clients. The deduplicatedsynthetic backup can be accessed by the backup system via the cloudspanning storage interface in the same manner as any other backup dataset. There is no additional overhead in assembling the synthetic backupfrom its label map and data segments as compared with any otherdeduplicated data stored in the cloud storage service.

FIG. 7 illustrates a computer system suitable for implementingembodiments of the invention. FIG. 7 is a block diagram of a computersystem 2000, such as a personal computer or other digital device,suitable for practicing an embodiment of the invention. Embodiments ofcomputer system 2000 may include dedicated networking devices, such aswireless access points, network switches, hubs, routers, hardwarefirewalls, WAN and LAN network traffic optimizers and accelerators,network attached storage devices, storage array network interfaces, andcombinations thereof.

Computer system 2000 includes a central processing unit (CPU) 2005 forrunning software applications and optionally an operating system. CPU2005 may be comprised of one or more processing cores. Memory 2010stores applications and data for use by the CPU 2005. Examples of memory2010 include dynamic and static random access memory. Storage 2015provides non-volatile storage for applications and data and may includefixed or removable hard disk drives, flash memory devices, ROM memory,and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other magnetic, optical, orsolid state storage devices.

In a further embodiment, CPU 2005 may execute virtual machine softwareapplications to create one or more virtual processors capable ofexecuting additional software applications and optional additionaloperating systems. Virtual machine applications can includeinterpreters, recompilers, and just-in-time compilers to assist inexecuting software applications within virtual machines. Additionally,one or more CPUs 2005 or associated processing cores can includevirtualization specific hardware, such as additional register sets,memory address manipulation hardware, additional virtualization-specificprocessor instructions, and virtual machine state maintenance andmigration hardware.

Optional user input devices 2020 communicate user inputs from one ormore users to the computer system 2000, examples of which may includekeyboards, mice, joysticks, digitizer tablets, touch pads, touchscreens, still or video cameras, and/or microphones. In an embodiment,user input devices may be omitted and computer system 2000 may present auser interface to a user over a network, for example using a web page ornetwork management protocol and network management softwareapplications.

Computer system 2000 includes one or more network interfaces 2025 thatallow computer system 2000 to communicate with other computer systemsvia an electronic communications network, and may include wired orwireless communication over local area networks and wide area networkssuch as the Internet. Computer system 2000 may support a variety ofnetworking protocols at one or more levels of abstraction. For example,computer system may support networking protocols at one or more layersof the seven layer OSI network model. An embodiment of network interface2025 includes one or more wireless network interfaces adapted tocommunicate with wireless clients and with other wireless networkingdevices using radio waves, for example using the 802.11 family ofprotocols, such as 802.11a, 802.11b, 802.11 g, and 802.11n.

An embodiment of the computer system 2000 may also include one or morewired networking interfaces, such as one or more Ethernet connections tocommunicate with other networking devices via local or wide-areanetworks.

The components of computer system 2000, including CPU 2005, memory 2010,data storage 2015, user input devices 2020, and network interface 2025are connected via one or more data buses 2060. Additionally, some or allof the components of computer system 2000, including CPU 2005, memory2010, data storage 2015, user input devices 2020, and network interface2025 may be integrated together into one or more integrated circuits orintegrated circuit packages. Furthermore, some or all of the componentsof computer system 2000 may be implemented as application specificintegrated circuits (ASICS) and/or programmable logic.

Further embodiments can be envisioned to one of ordinary skill in theart. In other embodiments, combinations or sub-combinations of the abovedisclosed invention can be advantageously made. The block diagrams ofthe architecture and flow charts are grouped for ease of understanding.However it should be understood that combinations of blocks, additionsof new blocks, re-arrangement of blocks, and the like are contemplatedin alternative embodiments of the present invention.

The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims.

1. A method of creating redundant data backups, the method comprising:receiving a backup data set and a backup metadata set from a backupsystem, wherein the backup metadata set is adapted to enable a backupsystem to restore data from the backup data set; transferring the backupdata set and the backup metadata set to a cloud data storage; connectinga recovery backup system to the cloud data storage; retrieving at leastthe backup metadata set from the cloud data storage; and updating therecovery backup system with the backup metadata set to enable therecovery backup system to restore data from the backup data set.
 2. Themethod of claim 1, comprising: restoring at least a portion of thebackup data set using the recovery backup system and the backup dataset.
 3. The method of claim 2, wherein restoring at least the portion ofthe backup data set is in response to receiving a user command.
 4. Themethod of claim 1, wherein: the backup data set comprises first backupdata and second backup data and the backup metadata set comprises firstbackup metadata and second backup metadata; and updating the recoverybackup system comprises updating the recovery backup system with thefirst backup metadata and second backup metadata.
 5. The method of claim4, wherein the recovery backup system is updated with the first backupmetadata and the second backup metadata in order of their creation. 6.The method of claim 1, wherein transferring the backup data set and thebackup metadata set to the cloud data storage comprises: deduplicatingat least a portion of at least one of the backup data set and the backupmetadata set; and transferring the deduplicated version of the portionof at least one of the backup data set and the backup metadata set tothe cloud data storage.
 7. The method of claim 6, wherein retrieving atleast the backup metadata set comprises: reconstructing anundeduplicated version of the backup metadata set from a deduplicatedversion of the backup metadata set stored in the cloud data storage. 8.The method of claim 1, wherein the backup metadata set includes a listthe data in the backup data set.
 9. The method of claim 1, wherein thebackup metadata set includes a log of backup operations consideredcomplete by the backup system.
 10. The method of claim 9, wherein thebackup operations include a backup replication.
 11. The method of claim9, comprising: providing the backup system with an indication ofcompletion of one of the backup operations upon transferring the backupdata set to the cloud data storage; accessing a log of the backup systemto create the log of backup operations included in the backup metadata;and transferring the backup metadata set to a cloud data storage. 12.The method of claim 9, wherein updating comprises: replaying the log ofbackup operations to the recovery backup system.
 13. A method ofrestoring data from a data backup set, the method comprising: connectinga recovery backup system to the cloud data storage; identifying a backupmetadata set stored in the cloud data storage; retrieving the identifiedbackup metadata set from the cloud data storage; updating the recoverybackup system with the backup metadata set to enable the recovery backupsystem to restore data from a backup data set; retrieving at least aportion of the backup data set from the cloud data storage; andrestoring at least the portion of data included in the backup data setusing the recovery backup system.
 14. The method of claim 13, whereinthe backup metadata set comprises a plurality of backup metadata, andupdating the recovery backup system with the backup metadata setcomprises: updating the recovery backup system with the plurality ofbackup metadata in the order of its creation.
 15. The method of claim13, wherein updating the recovery backup system comprises: determiningan order of creation for a plurality of backup metadata files, whereinthe plurality of backup metadata comprises the plurality of backupmetadata files.
 16. The method of claim 13, wherein retrieving at leasta portion of the backup data set from the cloud data storage comprises:reconstructing an undeduplicated version of the portion of the backupdata set from a deduplicated version of the backup data set stored inthe cloud data storage.
 17. The method of claim 13, wherein retrievingthe identified backup metadata set from the cloud data storagecomprises: reconstructing an undeduplicated version of the identifiedbackup metadata set from a deduplicated version of the backup metadataset stored in the cloud data storage.
 18. The method of claim 13,wherein restoring at least the portion of the data included in thebackup data set is in response to receiving a user command.
 19. Themethod of claim 13, wherein the backup metadata set includes a list ofavailable data in the backup data set.
 20. The method of claim 13,wherein the backup metadata set includes a log of backup operationsconsidered complete by a primary backup system.
 21. The method of claim20, wherein the backup operations include a backup replication by theprimary backup system.
 22. The method of claim 20, wherein the log ofbackup operations is generated following a transfer of the backup dataset from a primary backup site including the primary backup system tothe cloud data storage.
 23. The method of claim 20, wherein updatingcomprises: replaying the log of backup operations to the recovery backupsystem.